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ABSTRACT 

This evaluation was undertaken to analyse the overall merit of studies for publication in Medical Journals. Peer review, 
in which peer experts evaluate the value of a manuscript submitted to a professional journal, is regarded as a crucial step 
in publication. It helps to ensure that published articles describe experiments that focus on important issues and that the 
research is well designed and executed. By using previous guidelines and literature review, we have developed an 
assessment tool to evaluate the scientific studies in an effective and systematic order. Using these tools will facilitate 
comprehensive assessment and will contribute in generating constructive criticisms. 
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Introduction 

The need for qualified peer reviewers for 
biomedical journals has never been greater. 
Between 1966 and 1985, the number of journals 
and articles indexed by the National Library of 
Medicine increased by 30% and 73%, respectively 
(1). Sustained growth of biomedical literature and 
the staggering increase in drug research and 
development support the need for health 
professionals to take an active role in reviewing 
manuscripts submitted for pubUcation. The 
technical and scientific merits of manuscripts 
submitted for publication in a Journal are normally 
evaluated by an anonymous peer -review process. 
As with gathering the data, an important concern 
here is how long the data will take to analyse. The 
process of analysing and categorising quaUtative 
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data can be not only very time intensive but also 
voy subjective as one person may think it is almost 
perfect and another sees it riddled with problems. 
However, for the pubUsher, the evaluation process 
and its outcome is determined by some simple 
guidelines such as readability, subject matter (is it 
the kind of subject the publisher is looking for?), and 
marketability. Unfortunately, few formal training 
programs exist for those who are interested in 
developing their skills in this process (2) and there is 
limited formal assessment to analyse the peer 
reviewer performance (3). As with many aspects of 
clinical medicine, the adage "see one, do one, teach 
one" often applies to peer review. In this review we 
evaluate the present literature by generating an 
effective assessment tool for reviewing process. 

What is peer review? 

When a manuscript is submitted to a journal for 
consideration, the journal's editor is ultimately 
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responsible for its fate. If the manuscript is outside 
the journal's interest, the editor may reject it 
immediately or ask the author to resubmit it with 
revisions. If the editor considers the manuscript to be 
appropriate for the journal, the article will undergo 
internal review by members of the editorial advisory 
board and/or external peer review. External peer 
review is considered by most as a crucial part of 
manuscript evaluation; however, no consensus exists 
as to its definition. In one definition, peer review is 
"a negotiation between the author and journal about 
the scope of knowledge claims that will ultimately 
appear in print (4). Other interpretations of these 
definitions are less positive, with peer review being 
described as subjective, prejudicial, crude, offensive, 
and secretive (5). Harden's objective (6) structured 
clinical examination (OSCE) dramatically changed 
the assessment of professional competence to 
evaluate the performance of professional behaviours. 
There is a need for such a more comprehensive 
assessment tool to unify all these definitions on peer 
review (table 1) (7). 

Table 1. Dissecting the evaluation (AspinwaU 1992) 
Goal Why To generate an assessment tool for 
evaluation the value of a manuscript for 
publication 
Study design 

Using guidelines and cunent literature 
Where the study is performed 
During which period of time 



Aspect What 
Method How 
Sources Who 
Timing When 
Results So what? Constructive critics 



Evaluation Tools 

Although readers of the scientific literature must 
judge the quality of a research article for themselves, 
the peer-review system is an extremely valuable 
safeguard. First, it allows readers some degree of 
confidence regarding the quality of the article, which 
is particularly important in areas with which they are 
not familiar. Second, it reduces the time spent 
reading a paper that fails to conform to generally 
accepted standards. Thus, it is essential that 
reviewers carefully evaluate a manuscript, a process 



that often requires several hours. Each reviewer has a 
personal method of evaluating a manuscript, and a 
number of different approaches are described in the 
literature (8). A thorough evaluation should 
objectively judge all aspects of the manuscript as 
mentioned in table 2. Individuals should take the 
time necessary to a) thoroughly evaluate a 
manuscript they have agreed to review b) Consider 
the quality and significance of the experimental and 
theoretical work, the completeness of the description 
of methods and materials, the logical basis of the 
interpretation of the results, and the exposition with 
due regard to the maintenance of high standards of 
communication, c) Evaluations should include 
constructive suggestions for revision, including, rf 
appropriate, indication of where statements may 
require additional reference to the published 
literature. 



Table 2. Check list 


Sections 


Explanation 


Abstract 


How participants were allocated to 




interventions 


Objectives and 


Speciiic, Measurable, Achievable, Realistic 


Hypothesis 


and Time framed (SMART) 


Background 


Scientific background and explanation of 




rationale 


Participants 


Eligibility criteria for participants 


Sample size 


How sample size was determined 


Recruitment 


Dates defining the periods of recruitment 




and follow-up 


Baseline data 


Baseline demographic and clinical 




characteristics of each group 


Randomization 


Method used to generate the random 




allocation sequence. 


Statistical 


Statistical methods used to compare groups 


methods 


for primary outcome 


Methods 


Precise details of the interventions intended 




for each group and how and when they 




were actually administered 


Results 


Flow of participants through each stage 


Outcomes 


Clearly defined primary and secondary 
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outcome measures 



By using these tools and check list in evaluation, I 
have divided my observation and comments in major 
and minor. I had one major concern and a few minor 
with regard to the data interpretation (quality), the 
structure of this document, their methods and their 
results respectively. Finally a confidential cover letter 
to the editor accompanied in which briefly the 
scientific merit of the manuscript was summarized 
and appropriate advice was given. 

Table 3. Evaluation of scientific reliability of research 
using scoring criteria: 1-5, 1 indicating poor and 5 as 
excellent for each component of a study. Calculation of 
final scores will appear as excellent if it score between 
20-30 points, as average if between 10-20 and as poor 
if <10 scores 



Research question 


Clearly defined and appropriately 




answered? 


Overall design of 


Adequate, relevant 


study 




Participants 


Adequately described and their 


studied 


conditions defined? 


Methods 


Adequately described? Ethical 




issues discussed? 


Results 


Answer the research question? 




Credible? Well presented? 


Interpretation and 


Warranted by and sufficiently 


conclusions 


derived from/focused on the data? 




Message clear? 


References 


Up to date and relevant? Any 




glaring omissions? 



Evaluation process and outcomes 

Evaluation of research depends on complete and 
accurate reporting (9). In this section we briefly 
describe the assessment criteria. The technique of 
triangulation was used to generate reliable 
assessment criteria (10). Evaluation data were 
selected from various guidelines, and information for 
reviewers from various Medical Journals we worked 



with. The weaknesses should be identified by using 
assessment tools as summarized in table 2-3. 
Subsequently, the identified weaknesses might be 
divided into major as serious and in some cases the 
major might mean incompatible with further 
processing the manuscript leading to immediate 
rejection and minor weaknesses. The major concern 
is usually related to the quality and significance of 
the experimental and theoretical work. The 
manuscript finally might undergo an assessment 
according to a scoring system as indicated in table 3. 

Discussion 

Critical appraisal of the quality of clinical 
trials is possible only if the design, conduct, and 
analysis of these studies are thoroughly and 
accurately described in published articles (11). 
Some questions may arise during and after 
evaluation such as; does this work matter to 
clinicians, patients, teachers, or policymakers? 
Having this information, one can determine 
whether a given manuscript will meet the needs 
of the readers and be a suitable article for the 
journal? Therefore, it is also important to review 
the journal's mission and know its readership. 
Information about the journal can be gleaned 
from "the information for author's page" or from 
editorials written by the editor. This is essential 
in order to determine whether its content is 
within one's area of expertise of the journal; to 
establish whether one might have a conflict of 
interest; to identify the type of article; and to 
identify the study hypothesis or its objectives. 
Does the work add enough to what is already in 
the published literature? Does the paper read well 
and make sense (Table 3)7 

If the paper is suitable for the journal's reader 
with reasonable objectives and outcome, the 
critical read should be focused on two equally 
important questions: how relevant or important is 
this paper, and can the paper be improved, and if 
so, how? This step usually does not consist of a 
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single read-through. It may require a few reads to 
answer these questions properly. 

In the mid-1990s, two independent initiatives 
to improve the quality of reports of Randomized 
Controlled Trials (RCTs) led to the publication of 
the CONSORT (Consolidated Standards of 
/?eporting Trials) statement (12). This statement 
was developed by an international group of 
clinical trialists, statisticians, epidemiologists, 
and biomedical editors. Preliminary data indicate 
that the use of CONSORT does indeed help to 
assess the validity of RCTs results and to 
improve the quality of reports (13). They are 
primarily intended for use in writing, reviewing, 
or evaluating reports of simple two-group, 
parallel RCTs. 

As some other studies far from being 
transparent, the reporting of this and other related 
trials have been often incomplete, compounding 
problems arising from poor methodology (14). 
Other reasons for the authors' failure be 
screening strategy by using less reliable tools 
(methods) leading to undefined outcomes (Table 
2). On the other hand the search for accuracy and 
reliability must be balanced with the search for 
iimovation. Most agree that a flawed paper can 
have merit and it is important to not miss this by 
focusing too narrowly on quality control. 

At the end of this evaluation process raise the 
final question; whether the data provided by the 
authors are potentially publishable or not. It is 
not very straight forward and you need to come 
with a reasonable advice. It may be apparent that 
some papers will meet the rejection threshold, as 
they tend to meet this threshold in the methods 
section. Finally, the promise of qualitative 
research is more likely to be fulfilled if the health 
services research community more deeply 
understands the diversity of qualitative research 
and the methods criteria for evaluating it. Clear 
and effective criteria are critical for shaping a 
qualitative evaluation of a manuscript in as they 



provide a benchmark against which research can 
be assessed. 
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